SHANNON LECTURE Performance and Complexity

Our field has a rather remarkable history, in that it sprang almost fully fledged from the brow of one individual. In his original papers [1], Claude Shannon both posed the fundamental problems of information theory and also, to a large extent, answered them. Shannon's good news was this: for every memoryless channel there exists a parameter C, the channel capacity, such that the probability of error P r(E) can be made arbitrarily small for any rate R less than C by use of an appropriate code and decoder. Good codes exist, and in fact a randomly chosen code will with high probability turn out to be good. Shannon left a few loose ends, which have kept researchers occupied for nearly 50 years. In particular, his proof was nonconstructive, which left open the problem of finding specific good codes. Also, he assumed exhaustive maximum-likelihood decoding, whose complexity is proportional to the number of words in the code. It was clear that long codes would be required to approach capacity and therefore that more practical decoding methods would be needed. I arrived at M.I.T. in the early sixties, during what can be seen in retrospect as the first golden age of information theory research. (It is notable that over half of the previous Shannon Lecturers were associated with M.I.T. during that time.) Research at M.I.T. focused on two principal problems: how does P r(E) go to zero, and how does one practically approach capacity? The culmination of the first line of research was Gallager's elegant exponential error bounds for memo-ryless channels [2]. Gallager showed that the probability of error of the best block code of length N and rate R decreases exponentially with block length: P r(E) = exp ?NE(R); where the error exponent E(R) is greater than zero for all rates R less than the capacity C. A typical plot of E(R) vs. R for a very noisy channel, such as a power-limited Gaussian channel, is illustrated in Figure 1. The low-rate straight-line portion of the curve can be derived by a simple union bound; it has a slope of-1 and a value of R 0 at R = 0. Figure 1. Gallager's error exponent E(R) for a very noisy channel. Like Shannon, Gallager continued to assume a randomly chosen code and maximum-likelihood decoding. The decoding complexity G is then of the order of the number of codewords, G = exp …